Transformers and the Representation of Biomedical Background Knowledge

Abstract Specialized transformers-based models (such as BioBERT and BioMegatron) are adapted for the biomedical domain based on publicly available corpora. As such, they have potential to encode large-scale biological knowledge. We investigate encoding representation of knowledge in these models, its utility support inference cancer precision medicine—namely, interpretation clinical significance genomic alterations. compare performance different transformer baselines; we use probing determine consistency encodings distinct entities; clustering methods contrast internal properties embeddings genes, variants, drugs, diseases. show that do indeed knowledge, although some this is lost fine-tuning specific tasks. Finally, analyze how behave with regard biases imbalances dataset.